Skip to content

Spike/argocd demo #205

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
wants to merge 94 commits into
base: main
Choose a base branch
from
Draft

Spike/argocd demo #205

wants to merge 94 commits into from

Conversation

maltesander
Copy link
Member

@maltesander maltesander commented Apr 13, 2025

ArgoCD demo depolying stackable components and running a spark job via airflow.

What it does:

  • Stack ArgoCD

    • sealed secrets controller (https://github.com/bitnami-labs/sealed-secrets)
    • deploys all operators (except secret, listener and commons which are currently installed via stackablectl, since there must be operators installed in a stack) via applicationset
    • generated tls certs for sealed secrets
  • Demo based on the airflow demo, deployed via ArgoCD

    • deploy minio for airflow kubernetes executor logs
    • gitsync to fetch dags
    • automatically deploy minio connection to airflow
    • all secrets sealed
    • deploy spark job via UI

Clone the demos repo and checkout the branch spike/argocd-demo and run

stackablectl demo install argo-cd-git-ops -n argo-cd -s stacks/stacks-v2.yaml -d demos/demos-v2.yaml

or for custom repo to sync changes via argo (demo fork)

stackablectl demo install argo-cd-git-ops -n argo-cd -s stacks/stacks-v2.yaml -d demos/demos-v2.yaml --parameters customGitUrl=https://github.com/maltesander/demos --parameters customGitBranch=spike/argocd-demo

I currently port forward the argocd-server and airflow webserver. In the airflow webUI you can start the pyspark job manually.

Feedback would be very welcome :)

Edit - Notes:

  • Airflow does not offer https for the web ui currently. Thats the only part not properly encrypted in this demo.

@adwk67 adwk67 self-requested a review July 31, 2025 06:49
Copy link
Member

@adwk67 adwk67 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice job! Mainly just minor text formatting stuff and a few suggestions.

securityContext:
fsGroup: 1000
fsGroupChangePolicy: OnRootMismatch
runAsGroup: 1000
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This needs to be removed for the demo to run on Openshift. See stackabletech/nifi-operator#827
Ditto for runAsUSer.

Suggested change
runAsGroup: 1000

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done 03292c2

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

runAsUser needs to be removed, too. The demos are not primarily tested on Openshift, so maybe this point is moot. But maybe what is more relevant is to have a standard way of setting up Minio - the templated way is probably the way to go and so it would be good to have something we can copy/use between demos and tests (hence the suggested change).

@maltesander maltesander requested a review from adwk67 July 31, 2025 14:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants